1. Field of Invention
The present invention relates to the field of ultrasound imaging. More specifically, it relates to the stitching together of multiple ultrasound images.
2. Description of Related Art
Ultrasound imaging refers to the imaging of structures below a subject's surface (such as the imaging of internal human organs below a skin surface) by the sending of sound waves of known frequency into the interior of the subject and observing any sound waves that bounce back. By monitoring how long it takes for sound waves to bounce back from an internal structure, it is possible to estimate the depth and shape of the structure within the subject. It is also possible to discern some characteristics about the internal structure based on how the sound waves are absorbed, dispersed or deflected.
Ultrasonography, or diagnostic sonography, refers to the use of ultrasound imaging techniques for imaging subcutaneous body structures, or tissues, for diagnostic purposes. Ultrasound imaging may be used to image various types of tissues, such as muscle, fat, tendons, vessels, internal organs, etc. Another example is obstetric sonography, which is used to image a developing baby during pregnancy.
Ultrasound imaging typically applies a series of ultrasound waves at a frequency above the human audible range, and observed the sound waves that bounce back. Each sound wave is observed separately and constitutes a scan signal, or a scan line of an image. The collection of observed sound waves, or scan lines or scan signals, are placed sequentially next to each other to construct a two-dimensional image, in a manner similar to how images are created in cathode ray tube.
A problem with ultrasound images is that they are typically very noisy, due in part to the great many tissues and fluids of differing densities and types encountered by a sound wave as it propagates and dissipates through its downward and upward paths through an observed body.
Another problem with ultrasound images is that they are constructed by moving an ultrasound wand over the surface of a target tissue area, but the resultant ultrasound image formed from one pass of the ultrasound wand is typically very narrow. This provides a user (i.e., an ultrasound technician) with only a small observable part (or swatch or slice) of the whole of the target tissue area. As a result, multiple swatches are typically needed to gather enough imaging information to span the whole of the target area. That is, a technician must make multiple passes with the ultrasound wand, store the image information from each pass, and try to put together the image information from the different passes.
The ability to stitch together multiple ultrasound images from multiple passes to create on larger ultra sound image is therefore beneficial. To stitch images together refers to the combining of image information from two or more images as seamlessly as possible/practical. Much has been written in the field of stitching digital camera images to create panoramic views, but many of the techniques used to stitch digital camera images rely on the images being of good quality. Unfortunately, ultrasound images are very noisy and often “dark”, which limits the application of typical image stitching techniques. Another difficulty is that ultrasound images are not two-dimensional arrays of pixels, as is the case of digital camera images, but are rather individual, one-dimensional, scan lines that are typically processed individually. This further limits the porting of image stitching techniques from the field of digital camera imaging systems to the field of ultrasound imaging systems.
Nonetheless, there are several examples of stitching ultrasound images. One example is found in European patent EP1531730A1 to Chin et al, which describes the stitching of multiple ultrasound images to construct a composite whole for aid in the diagnosis of breast cancer. Another example is provided in “Rapid Image Stitching and Computer-Aided Detection for Multipass Automated Breast Ultrasound”, Med. Phys. 37 (5), May 2010, by Chang et al., which describes using the sum of absolute block-mean difference (SBMD) measure to stitch ultrasound images.
Stitching multiple ultrasound images has some self-evident advantages, but image stitching in general (such as that used in digital camera images) is an independent field of study. For example, images of a scene may be stitched together to create a paranormal view, as is described in “Generating Panorama Photos”, Proc. of SPIE Internet Multimedia Management Systems IV, vol. 5242, ITCOM, Orlando, September 2003, by Deng et al. Another example of image stitching is found in U.S. Pat. No. 8,319,823 to Chen et al.
In general, image stitching requires two more adjacent images having some overlapping portion. Characteristic features of each image (at least within their overlapping portions) are identified and described. The distinctive descriptions of the characteristic features in one image are then compared with those of its adjacent image to identify characteristic features that may correspond to each other (and thus correspond to the same point on an imaged scene). Characteristic features that correspond to each other may be said to be “indexed” to each other. In this manner, an index of corresponding (i.e. matched or correlated) characteristic features in the overlapping portions can be established, and this indexing is then used to align and stitch together the two images.
For example in FIG. 1, images 7A, 7B, 7C and 7D each provide partial, and overlapping, views of a building in a real-world scene, but none provide a full view of the entire building. However, by applying characteristic feature detection and indexing (i.e. identifying matching pairs of) characteristic features in the four partial images 7A, 7B, 7C and 7D that correlate to the same real feature point in the real-world scene, it is possible to stitch together the four partial images to create one composite image 7E of the entire building.
Preferably, the characteristic features are uniquely recognizable and distinctively describable. The characteristic features may includes edges, corners, blobs (or regions) and/or individual pixels (i.e. points). When the characteristic features are corners, they are often termed “interest points”. The Harris corner detector is a well-known mechanism/method for identifying interest points (i.e. characteristic corners). When the characteristic features are individual points, such as individual pixels, they are often termed “feature points”. The SIFT (or affine SIFT) algorithm is a well-known mechanism/method for identifying feature points. It noted that a corner is a point (i.e. a pixel) and although the Harris corner detector was initially developed to find corners, it was soon realized that the definition of “corner” permitted it to uniquely identify individual points (i.e. pixels). Thus, like the SIFT algorithm, the Harris corner detector also uniquely identifies/describes individual distinctive pixels, but may be more selective than the SIFT algorithm since it limits itself to pixels having characteristics distinctive of corners. It is further noted that the individual points identified by the Harris corner detector are still commonly termed “interest points” instead of “feature point” for historical reasons. Nonetheless, both algorithms are popular and well-known methods of identifying characteristic features (i.e. interest points and feature points).
In the present discussion, however, the terms “interest point” and “feature point” will be used interchangeably to generally refer to a characteristic feature, or a point of interest, i.e. an image point that can be distinctively identified and described within an image (or within a window area within an image). Thus, interest point and feature point may both refer to a characteristic pixel within an image.
Because ultrasound images are generally narrow, and multiple overlapping imaging passes of an ultrasound wand are required to span the breath of a target object of observation, it is an object of the present invention to provide automated stitching of the many overlapping ultrasound images.
It is also an object of the present invention that the automated method be robust enough not to be confused by the large amount of noise found in ultrasound images.
Since the ultrasound images may be taken in any direction that the ultrasound imaging wand is moved, a further object of the present invention is that the automated system be capable of automatically identifying the overlapping portions of the different images and automatically orient the different image correctly for stitching.
It is a further object of the present invention that the automated stitching system post-process the stitched image so as to produce relatively seamless stitching with consistent imaging tones across the composite, stitched image.